It is a full-time job staying on top of all the AI/LLM developments. For my role that means implementing new research and tooling to make our products more secure and safe to deploy. Promptfoo is one of those tools that has landed on my radar as a no-frill entry point into AI red teaming.
Why Red Team AI?
Red teaming for AI is about finding new ways to bypass the safeguards put in place to prevent the model from generating harmful content. LLM red teaming is an ongoing effort to identify vulnerabilities in AI models and to develop mitigations to prevent them from being exploited. The goal is to ensure that AI models are safe, secure, and reliable. It is a constantly evolving field as the models themselves are constantly evolving.
Shared AI Responsibility
My cyber background is almost entirely cloud security. When I started learning about AI/LLM security tools I kept getting confused on why tools needed to know what model I was using. Hadn't the frontier models (Claude, OpenAI, etc.) already been red-teamed and secured? I was under the impression that the responsibility was on the model developers to ensure their models were secure.
I mention my background in cloud security because this is how AI security finally clicked.
With AWS, the line is "security of the cloud" vs. "security in the cloud". Amazon secures the hardware, the hypervisor, the physical data center. You secure your VPC config, your IAM policies, your S3 bucket permissions, your app. AWS handling you a secure hypervisor doesn't save you from leaving a bucket public - that part's on you.
LLMs have the same seam:
The model provider (Anthropic/OpenAI) secures the model itself - the "of the model" part. They do the alignment training, the safety evals, the guardrails baked into the weights, the refusal behavior, the infrastructure the model runs on. Thats their side, and you can't touch it or fix it.
You secure everything you build around the model - the "in your application" part. Your system prompt, your RAG pipeline and what data it pulls in, your MCP tool wiring and scopes, your input/output handling, your authn/authz around the chatbot, what you log and how you protect it. All yours.
And the key insight carries over exactly: a secure foundation doesn't make your layer secure. Same wat a hardened hypervisor doesn't help a public s3 bucket.
However, the line moves depending on how you consume the model. Just like AWS shifts responsibility as you go with EC2, ECS Lambdas managed service, your AI responsibility shifts with the integration pattern. RAW API call = you own the most (all the wrapping is yours). A more managed "agent platform" or a vendor hosted RAG = they take on more of the middle layer. Bedrock is the cleanest parallel - AWS then shares some of the model-hosting and guardrail layers with you, but your prompts, data and tool config are still yours. Worth naming which pattern each of your systems uses, becauses it tells you exactly where your responsibility starts.
One caveat, unlike AWS, where the provider side is genuinely opaque-but trusted, the model layer can still fail in ways that become your problem operationally - a model update changes refusal behavior or a jailbreak class the vendor hasn't patched gets through. You can't fix it, but you're still the one whose chatbot misbehaves in front of a user. Thats the argument for runtime-monitoring loop; it's how you catch the provider's side degrading, even through remediation means a support ticket and a guardrail on your end rather than a code fix.
So what are you actually testing? ELI5 version
Think of the frontier model (Claude, GPT, etc) as a car engine. Anthropic and OpenAI build the engine. They test the engine before selling it - crash tests, emissions, the works.
Say you have a chatbot, this is your car you built around that engine. You added the steering wheel (your system prompt), the seats and dashboard (your UI), a GPS that pulls from your own maps (your RAG/knowledge base), and maybe it can open the garage door and start the coffee maker (your MCP tools).
The engine beind safe does not mean your car is safe. You could have wired the coffee maker button so anyone honking the horn triggers it. That bug is yours, not the engine's.
So now you understand we're testing the car, not the engine. AI security tools don't care whether they're hitting a raw engine or a whole car. They send messages to an endpoint and see what comes back.
Point your tools at your chatbot's endpoint (the URL your app talks to, or the chat UI itself) now every probe goes through your system prompt, your RAG, your tools before it reaches the engine and comes back. This is what you want. Remember you're testing the car.
Why testing your car finds different bugs than testing the engine
When a jailbreak probe hits your chatbot, it's not really asking "can I break Claude?" It's asking things you actually care about:
Can I make your bot reveal its system prompt? (Now I know your internal rules.) Can I trick your bot into calling an MCP tool it shouldn't? (Confused-deputy - your agentjacking worry.) Can I get your bot to spit out another user's data that your RAG pulled in? (Sensitive data leak - the engine never knew that data existed; your car fed it in.)
The engine passing its own safety tests tells you nothing about these, because these bugs live in the parts you built.
Once I understood that I wasn’t testing Claude—I was testing everything I’d built around Claude—the tooling started making sense.